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I.  INTRODUCTION 


Conditions  Which  Prompted  this  Study 

On  December  29,  1970,  Public  Law  91-596,  the  Williams-Steiger  Occupational 
Safety  and  Health  Act  (OSHA)  was  signed  into  law  by  the  President  of  the  United 
States.  Under  its  provisions,  employers  are  required  to  provide  a  safe  and 
healthful  working  environment  for  ail  employees.  Since  the  Act's  passage,  a 
number  of  subsequent  Federal  directives  and  standards  which  mandate  establish¬ 
ment  of  an  occupational  safety  and  health  program  for  Federal  employees  and 
define  the  program's  scope  have  been  issued.  Army  Regulation  (AR)  40-5,  Health 
and  Environment,  describes  the  Army  Occupational  Health  Program.  The  program  is 
applicable  to  all  Department  of  the  Army  commands  and  includes  the  following: 

1.  Evaluation  and  control  of  the  health  hazards  and 
physical  stresses  of  the  work  environment  and  promotion 
of  necessary  measures  to  assure  the  safety  of  workers 
within  that  environment; 

2.  Provision  of  initial  and  periodic  medical  examinations 
related  to  job  hazards  and  physical  requirements  to  assure 
that  all  workers  are  physically,  mentally  and  psychologically 
suited  to  their  work; 

3.  Provision  of  medical  care  for  occupational  illnesses  and 
injuries,  and  emergency  and  limited  palliative  care  for 
nonoccupational  illnesses  and  injuries  with  referral  to  the 
workers'  personal  physicians; 

4.  Provision  of  job-related  and  general  health  counseling 
and  education; 

5.  Conduct  of  health  maintenance  activities  such  as  job- 
related  surveillance  of  pregnant  employees  and  personnel 
with  chronic  disabilities,  medical/nursing  support  of  the 
sickness  absence  control  program,  immunization  programs, 
and  voluntary  health  examinations  or  disease  screening 
programs ; 

6.  Maintenance  of  medical  records  and  reports; 

7.  Analysis  and  interpretation  of  statistics  and  services. 1 
Each  Army  installation  is  required  to  have  an  ongoing  occupational  health 
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program  for  assigned  civilian  and  military  personnel.  Installation  programs  are 
evaluated  for  compliance  with  legal  and  regulatory  requirements  by  personnel 
assigned  to  the  U.S.  Army  Environmental  Hygiene  Agency  (USAEHA)  approximately 
every  three  years.  A  detailed  Occupational  Health  Program  Survey  Questionnaire 
is  sent  to  each  installation  for  completion  in  advance  of  the  survey.  The  data 
which  is  provided  is  subsequently  reviewed  on-site.  Several  survey  team  members 
have  expressed  a  degree  of  dissatisfaction  with  the  current  evaluation  methodo¬ 
logy  employed,  which  is  felt  to  be  somewhat  subjective.2  A  better  evaluation 
technique  has  not,  however,  been  developed. 

A  related  situation  exists  with  respect  to  occupational  health  program 
evaluation  at  the  installation  level  in  the  interim  between  USAEHA  surveys.  At 
the  present  time,  no  formal  evaluation  is  being  conducted  by  medical  personnel 
at  Fort  Devens  or  at  any  of  the  five  health  clinics  on  Army  installations  in  New 
York  and  Massachusetts  for  which  the  Fort  Devens  MEDDAC  is  responsible.  The 
lack  of  an  established  occupational  health  program  evaluation  methodology  at 
Fort  Devens  was  revealed  during  an  August,  1982  occupational  health  survey  of 
the  installation. ^ 


The  Problem 

The  problem  addressesd  by  this  study  is  to  test  a  proposed  methodology  for 
evaluating  an  occupational  heath  program  at  a  U.S.  Array  installation. 

Factors  Influencing  the  Proposed  Methodology  to  be  Tested 
Army  Occupational  Health  Program  factors 

The  content  of  Array-published  docunents  which  pertain  to  the  Army 
Occupational  Health  Program  and  the  ways  in  which  installation  occupational 
health  programs  are  normally  administered  have  greatly  impacted  upon  the  selec¬ 
tion  of  the  test  methodology.  AR  40-5  charges  the  commander  of  the  medical 
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facility  which  provides  medical  support  to  each  installation  with  the  respon¬ 
sibility  for  providing  technical  guidance  and  direction  to  the  installation 
occupational  health  program.  This  task  is  generally  delegated  to  the  Medical 
Department  Activity  (MEDDAC)  Preventive  Medicine  Activity. ^ 

Health  Services  Conmand  (HSC)  Regulation  11-4  specifically  directs  the 
Chief,  Preventive  Medicine  Activity,  to  assess  the  overall  occupational  health 
program. 5  On  an  installation  which  is  served  by  a  health  clinic  that  is  subor¬ 
dinate  to  a  MEDDAC  located  elsewhere,  the  health  clinic  staff  performs  most 
occupational  health  program  functions.  Primary  responsibility  for  program  evalu 
ation  in  this  setting  rests  with  the  occupational  health  physician  and/or 
occupational  health  nurse. 6 

USAEHA  Technical  Guide  (TG)  124,  Occupational  Health  Program  Manual,  states 
that  "Ongoing  and  periodic  (later  referred  to  as  annual)  evaluation  is  essential 
to  having  an  effective  and  efficient  (occupational  health)  service."7  The  guide 
also  points  out  the  primary  purposes  of  program  evaluation,  which  are  the  docu¬ 
mentation  of  what  is  being  accomplished,  any  unmet  needs,  and  program 
weaknesses.  Evaluation  techniques  to  be  employed  are  not  specified;  however, 
what  jjj  emphasized  is  the  need  to  assess  compliance  with  provisions  of  the 
overall  Army  Occupational  Health  Program  and  performance  relative  to  specific 
installation  programs.  The  guide  also  discusses  the  necessity  for  development  of 
objectives  and  standards  against  which  program  performance  should  be  measured 
and  the  need  to  develop  criteria  and  identify  data  elements  to  facilitate  the 
measurement  process.  The  desirability  for  program  evaluation  to  be  a  coor¬ 
dinated  activity  among  occupational  health  service  staff  members,  including 
occupational  health  physicians  and  nurses,  is  additionally  stressed. 8 

The  occupational  health  nurse  plays  a  key  role  in  the  evaluation  process. 
The  duties  of  this  individual  often  include  the  maintenance  of  lists  of 
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installation  personnel  potentially  exposed  to  specific  occupational  health 
hazards,  conduct  of  many  routine  health  screening  examinations,  organization 
and  presentation  of  health  education  programs,  and  compilation  of  occupational 
illness  and  injury  data.  USAEHA  TG-005,  Occupational  Health  Nursing 
Responsibilities,  lists  program  evaluation  responsbilities  which  the  pro¬ 
fessional  nurse  can  assume  in  order  to  contribute  most  effectively  to  the 
accomplishment  of  the  occupational  health  service  mission.  These  are  stated  as 
follows: 


1.  Evaluates  health  programs,  nursing  procedures,  and 
responsibilities  and  recommends  methods  that  will 
increase  effectiveness; 


2.  Conducts  periodic  audit  of  medical  and  nursing  programs, 
procedures,  and  records,  and  recommends  changes  as 
indicated. ^ 


The  Army  Occupational  Health  Program  is  broken  down  into  distinct  activities 
or  program  elements.  These  may  be  required  by  law  and/or  regulation  or  are 
considered  desirable  for  implementation  as  resources  permit.  A  listing  of  Army 
Occupational  Health  Program  Elements  is  presented  in  Appendix  A.  The  organiza¬ 
tion  of  overall  program  requirements  by  element  is  useful  to  the  proposed 
evaluation  process. 


Quality  assurance  factors 

In  April  1979,  the  Board  of  Coninissi oners  of  the  Joint  Canmission  on 
Accreditation  of  Hospitals  (JCAH)  approved  a  new  Quality  Assurance  (QA)  stan¬ 
dard.  The  new  standard  was  first  published  in  the  1981  JCAH  Accreditation 
Manual  for  hospitals  and  became  effective  on  January  1st  of  that  year. 
Implementation  of  the  standard  is  now  widespread  throughout  the  hospital 
industry. 

The  objective  of  the  JCAH  in  formulating  and  requiring  hospital  compliance 
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with  the  quality  assurance  standard  is  the  enhancement  of  patient  care.  The 
standard  mandates  that  hospitals  develop  mechanisms  for  assessing  patient  care 
acti  ..ties.  The  wording  of  the  standard  is  intentionally  broad  so  as  not  to 
limit  innovation  in  the  choice  of  assessment  alternatives.  It  is  clear, 
however,  that  any  assessment  methodology  which  is  selected  must  focus  on  the 
correction  of  patient  care  problems  which  are  identified  by  comparing  actual 
performance  with  written,  pre-established,  clinically  valid  criteria. 10  These 
are  defined  as  predetermined  elements  against  which  aspects  of  the  quality  of 
medical  service  may  be  compared. H 

The  scope  of  JCAH  quality  assurance  interest  encompasses  patient  care  acti¬ 
vities  in  both  inpatient  and  ambulatory  care  settings.  Hospital  quality 
assurance  programs  are  currently  being  closely  scrutinized  by  JCAH  surveyors. 

Because  of  the  emphasis  which  is  currently  being  placed  on  the  proper  conduct  of 
quality  assurance  activities  by  the  JCAH,  the  subject  is  of  special  interest  to 
the  Army  Surgeon  General.  Health  Services  Command  Inspector  General  (IG)  teams, 
like  JCAH  surveyors,  therefore  look  at  quality  assurance  quite  closely  during 
their  annual  general  inspections  of  Army  MEDDACS,  medical  centers,  and  subor¬ 
dinate  health  clinics.  It  is  during  these  inspections  that  performance  with 
respect  to  the  installation  occupational  health  program  is  most  likely  to  be 
reviewed.  At  many  installation  health  clinics  where  occupational  health  activi¬ 
ties  represent  the  bulk  of  the  patient  care  workload,  quality  assurance  and 
occupational  health  may  become  closely  intertwined. 

Linking  occupational  health  program  evaluation  with  quality  assurance 

Much  commonality  of  purpose  exists,  then,  in  conducting  installation  occu¬ 
pational  health  program  evaluation  and  quality  of  care  assessment.  The  guidance 
which  has  been  published  for  accomplishing  each  of  these  reviews  is  likewise 
similar. 
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Considerable  overlap  of  the  two  programs  may,  in  fact,  exist.  It  is  therefore 
logical  to  apply  concepts  and  processes  which  have  been  used  for  assessing 
quality  of  care  to  the  evaluation  of  an  installation  occupational  health 
program. 


A  Review  of  the  Literature 
Occupational  health  and  quality  assessment 

A  relationship  between  quality  assessment  and  program  evaluation  is, 
according  to  Donabedian,  established  when  a  given  program  includes  the  provi¬ 
sion  of  personal  health  services. ^  If  a  program  encompasses  other  functions  as 
well,  then  quality  assessment  becomes  part  of  overall  program  evaluation.  The 
importance  of  quality  assessment  relative  to  overall  program  evaluation  is 
increased  in  situations  where  the  medical  authority  is  considered  to  be  respon¬ 
sible  for  a  program  involving  a  specified  enrolled  population.  In  this 
situation,  the  distinction  between  the  two  concepts  may  become  somewhat 
blurred. 13 

There  is  little  in  the  literature  which  specifically  discusses  quality 
assurance  activities  in  relation  to  occupational  health.  One  exception  is  an 
editorial  published  in  the  April  1978,  Journal  of  Occupational  Medicine,  in 
which  Dr.  John  H.  Mitchell  presents  a  case  for  the  performance  of  ambulatory 
medical  audits  of  the  quality  of  occupational  medical  care.  He  argues  that  a 
need  exists  to  identify  delivery  of  occupational  medical  care  problems  and  ini¬ 
tiate  corrective  actions. 14 

Occupational  health  program  evaluation 

Several  articles  have  addressed  methodologies  for  occupational  health 
program  evaluation.  Davis  discussed  an  audit  procedure  of  use  to  an  organization 
which  desires  an  objective  appraisal  of  its  occupational  health  activities. 13 

Krieger,  in  an  unpublished  manuscript,  proposed  a  corporate  occupational  medicine 
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audit  methodology  which  is  based  on  an  evaluation  of  discrete  program  elements. 
His  concept  called  for  the  generation  of  performance  criteria  and  measurement 
tools,  but  did  not  go  into  detail  on  how  these  would  be  developed.  The  need  for 
quality  of  care  assessment  was  recognized,  with  quality  assurance  being  con¬ 
sidered  as  a  distinct  activity  in  the  overall  audit  process. 16 

The  most  exhausive  work  to  date  on  occupational  health  program  evaluation 
was  performed  by  a  University  of  Cincinnati  group  in  1975.  Their  efforts  culmi¬ 
nated  in  the  publication  of  the  document,  Standards,  Interpretations,  and  Audit 
Criteria  for  Performance  of  Occupational  Health  Programs,  the  contents  of  which 
are  used  as  a  basis  for  occupational  health  program  accreditation  by  the 
Occupational  Health  Safety  Program  Accreditation  Commission  (OHSPAC).  Broad, 
goal-oriented  standards  describing  optimally  desired  levels  of  performance  for 
various  program  areas  were  developed  and  validated  along  with  assessment  criteria 
formatted  as  audit  questions.  The  validation  process  encompassed  a  thorough 
review  by  numerous  occupational  health  experts  in  order  to  ensure  professional 
acceptability.  Although  the  document  is  considered  to  be  useful  in  the  occupa¬ 
tional  health  program  assessment  process,  its  authors  acknowledge  that  portions 
of  the  document  may  not  be  applicable  to  all  programs  due  to  differences  in 
program  size  or  activities. 1?,18 

Quality  assessment  concepts 

The  JCAH  quality  assurance  standard  links  the  quality  of  performance  with 
the  degree  of  adherence  to  pre-established,  or  explicit  criteria  of  care.  These 
are  commonly  divided  into  three  major  categories.  ’’Structural"  criteria  are 
those  which  focus  on  the  tools  and  resources  which  providers  have  at  their 
disposal  and  the  settings  in  which  care  is  delivered. 19  Process  criteria  examine 
the  practice  of  medicine,  or  patient  management  activities. 20  "Outcome"  criteria 
concern  themselves  with  the  end  result  of  care, or  what  actually  happens  to  the 
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patient.  Strengths  and  weaknesses  of  each  of  the  above  categories  have  been 
extensively  reviewed  in  the  literature.  While  a  general  consensus  exists  that 
quality  assessment  should,  insofar  as  possible,  be  related  to  the  measurement  of 
the  results  of  care,  outcome- related  data  is  often  available  only  in  the 
inpatient  setting.  Information  necessary  to  evaluate  ambulatory  patient  care 
from  an  outcome  standpoint  is  generally  not  recorded  in  patient  charts.  For 
this  and  other  reasons,  most  ambulatory  evaluation  methodologies  are  process- 
oriented. 

Donabedian  defends  the  use  of  process  criteria  in  quality  assessment  on  the 
basis  that  certain  procedures  used  in  specified  situations  are  clearly  associated 
with  good  results. 22  Griffith  is  somewhat  less  supportive  but  does  concede  that 
process  criteria  do  identify  specific  events  which  can  be  studied  and 
corrected.23  Neither  is  particularly  satisfied  with  the  use  of  structural 
criteria  because  of  insufficient  knowledge  about  relationships  between  structure 
and  performance. 24 > 23 

The  assessment  of  preventive  health  services  in  the  ambulatory  care  setting 
must,  in  the  absence  of  observable  or  measureable  changes  in  health  status,  be 
accomplished  largely  through  the  use  of  criteria  relating  to  structure  or  pro¬ 
cess.  The  few  articles  on  this  subject  which  appear  in  the  literature  concern 
themselves  primarily  with  maternal  and  child  care  evaluation.  In  each  case, 
preventive  assessment  criteria  were  heavily  process-oriented. 26, 27, 28  jn  one 
study,  participants  who  were  asked  to  support  their  selections  of  criteria  with 
docunentation  from  the  literature  offered  few  references,  a  finding  which  indi¬ 
cates  a  lack  of  research  to  establish  the  value  of  many  procedures  customarily 
performed  during  well-child  visits. 29 
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icit  versus  implicit  criteria 

Considerable  research  has  been  devoted  to  the  development  of  medical  care 
assessment  criteria.  These  may  be  categorized  as  implicit  or  explicit. 

Implicit  criteria  are  subjective  elements  related  to  the  appropriateness  of  care 
which  are  based  on  the  personal  opinion  of  each  assessor.  Explicit  criteria,  on 
the  other  hand,  are  elements  of  evaluation  which  are  agreed  upon  by  a  group  and 
set  down  on  paper  prior  to  the  conduct  of  the  assessment  process.  Lists  of 
explicit  criteria  have  been  formulated  from  the  literature  and  by  committees  of 
individuals. 

The  advantages  and  limitations  to  the  application  of  implicit  and  explicit 
criteria  in  medical  care  quality  assessment  have  been  widely  discussed. 30,31, 32, 33 
Conments  have  focused  upon  issues  of  reliability,  validity,  ease  of  assessment 
conduct,  and  the  desirability  for  fairness,  consistency,  and  measureability. 

The  current  trend,  as  evidenced  by  the  wording  of  the  JCAH  Quality  Assurance 
Standard  and  the  definition  of  the  term  "criteria"  published  in  the  PSRO  Program 
Manual  (and  previously  included  herein)  is  clearly  toward  the  use  of  explicit 
criteria  as  quality  of  care  screening  devices.  The  U.S.  Army  has  followed  the 
lead  of  the  JCAH  in  requiring  the  use  of  predetermined,  written  criteria  in 
patient  care  assessment.-^ 

Process  criteria  validity  and  reliability 

The  validity  and  reliability  of  process  criteria  are  of  paramount  importance 
to  those  seeking  to  devise  appropriate  quality  of  care  screening  elements. 

Validity  concerns  itself  with  the  relevance  of  criterion  performance  to  outcome, 
while  reliability  measures  the  extent  of  agreement  on  the  importance  of  a  given 
criterion  to  quality.  Unfortunately,  experimental  evidence  verifying  the 
validity  of  process  criteria  is  often  lacking.  Therefore,  criteria  accep¬ 
tability  must  ,  in  many  cases,  be  based  on  a  consensus  among  knowledgeable 
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individuals  as  to  relevance. 

A  nunber  of  studies  conducted  to  determine  process  criteria  validity  appear 
in  the  literature.  Investigations  generally  consisted  of  two  parts:  initial 
development  of  diagnosis-specific  assessment  criteria  having  the  greatest  con¬ 
sensual  agreement  (reliability)  among  experts,  and  subsequent  comparison  of 
these  group  developed  assessment  criteria  to  published  data  correlating  process 
and  outcome  or  to  observed  patient  out comes. 5 5, 36, 37  jn  one  Qf  these  studies, 
Dershewitz  et  al.  identified  validation  data  from  the  literature  for  ten  of  the 
thirteen  peptic  ulcer  diagnostic  criteria  most  often  cited  in  predeveloped 
listings  compiled  by  various  professional  groups. 3**  Hastings,  et  al.  constructed 
and  validated  a  checklist  used  in  the  performance  of  peer  review  of  ambulatory 
medical  records.  A  checklist  of  criteria  weighted  according  to  perceived  impor¬ 
tance  was  first  developed  by  a  panel  of  academic  clinicians.  A  series  of  iden¬ 
tical  medical  records  were  then  reviewed  against  the  checklist  by  different 
individuals  in  order  to  establish  reviewer  reliability.  Validity  was  tested  by 
reexamination  of  ten  jail  inmate  patients  by  a  specially  trained  reevaluation 
physician  who,  using  the  checklist,  evaluated  the  quality  of  care  which  these 
patients  had  orginally  received  at  an  earlier  sick  call. 59 

McAuliffe,  in  a  critique  of  process- out come  correlation  studies,  notes 
additional  investigation  results  which  appear  to  validate  process  data. 40  He 
also  brings  to  light  flaws  in  the  conduct  of  five  studies  in  which  little  rela¬ 
tionship  was  found  between  process  and  outcome. 41  He  concludes,  however,  that 
very  little  is  currently  known  about  the  validity  of  methods  being  used  to 
assess  quality  of  care. 42 

Studies  of  reliability  have  focused  on  measuring  consensus  among  pro¬ 
fessionals  as  to  what  criteria  are  important  to  quality  of  care  assessment 
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and  their  relative  levels  of  importance.  As  previously  stated,  the  degree  of 
consensus  is  often  accepted  as  evidence  of  the  relationship  between  process  and 
outcome.  Donebedian  notes  that  Brook,  in  a  study  comparing  five  methods  of  peer 
review,  reported  a  high  degree  of  correlation  between  sets  of  criteria  endorsed 
by  at  least  two-thirds  of  separate  groups  of  specialists  and  generalists.45 
Hare  and  Barnoon  echo  this  finding  in  reporting  high  levels  of  agreement  between 
academic  and  practicing  internists  on  the  relative  importance  of  criteria  deve¬ 
loped  for  six  categories  of  care.44  In  yet  another  study,  Osborne  and  Thompson 
report  a  significant  concordance  among  academicians  and  practitioners  on  the 
relevance  of  criteria  pertaining  to  seven  pediatric  health  problem  areas.  It 
should  be  noted  that  the  greatest  disagreement  in  relevance  ratings  occurred  in 
lists  of  criteria  developed  to  assess  well  child  care  in  four  distinct  pediatric 
age  groupings.45 

Study  results  reported  above  should  not  be  accepted  as  conclusive  evidence 
of  universal  concordance  on  criteria  among  professionals  of  the  same  specialty. 
Some  significant  differences  of  opinion  did,  in  fact,  occur,  but  these  tended  to 
be  overshadowed  by  overall  findings.  It  is  difficult,  then,  to  proffer  any  firm 
conclusions  on  the  matter  of  reliability. 

Other  characteristics  of  explicit  process  criteria 

In  addition  to  the  desirability  for  criteria  to  be  considered  valid  (often 
an  assunption  based  on  the  results  of  reliability  studies),  several  other  quali¬ 
ties  of  criteria  should  be  considered  during  the  development  process.  Army 
Regulation  40-66  mandates  that  criteria  be  realistic,  objective,  and 
measureable.  The  term  "realistic”  addresses  the  need  for  feasibility  in  meeting 
a  given  criterion.  For  example,  it  is  useless  to  establish  a  criterion  calling 
for  a  computer  analysis  of  data  if  a  computer  is  not  available  to  perform  this 
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function.  Objectivity  infers  precision  and  detail;  a  criterion  should  be  rela¬ 
tively  inmine  to  varying  interpretation  by  different  individuals. 46  Merry 
suggests  that  the  most  desirable  criterion  is  the  one  that  requires  an  indivi¬ 
dual  to  exercise  ver*  little  judgement  and  answer  its  implicit  question  with  a 
relatively  straight  ..orward  yes  or  no  response. 47  Measureability  is  defined  as 
the  requirement  for  the  answer  to  a  given  criterion's  implicit  question  to  be 
verifiable  by  observation,  consultation,  or  docunentation. 

Acceptability  of  criteria  overlaps  with  validity  and  infers  the  existence 
of  a  consensus  between  knowledgeable  individuals  as  to  the  value  of  a  given  cri¬ 
terion.  A  criterion  may  be  considered  to  be  valid,  but  it  may  duplicate  another 
criterion  in  the  same  list,  be  considered  low  in  importance,  or  may  possess  some 
other  characteristic  which  would  deem  it  to  be  unacceptable  as  an  assessment 
element . 

Techniques  for  criteria  derivation 

The  collective  opinion  of  a  group  of  experts  is  often  sought  through  con¬ 
sultation  techniques  which  facilitate  the  formation  of  a  group  consensus.  This 
goal  may  be  accomplished  by  questioning  group  members  separately  or  via  a  face- 
to-face  panel  format. 

The  Delphi  technique  is  well  known  to  researchers  seeking  to  reduce 
variation  in  expert  opinion.  Its  application  to  criteria  derivation  involves 
the  modification  of  criteria  lists  based  on  mailed  feedback  received  from 
separate  panelists.  In  two  studies  employing  the  Delphi  technique,  previously 
established  criteria  lists  were  originally  submitted  for  first-round  panelist 
assessment.  After  initial  consolidation  of  first-roisid  results  by  researchers, 
each  panelist  was  provided  with  a  sumnary  of  the  group  response  and  was  afforded 
the  opportunity  to  alter  his  opinion.  In  each  case,  opinion  modification 
resulting  from  the  second-round  mailing  was  minimal. 48, 49, 50, 51 
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A  face-to-face  group  process  which  has  seen  application  in  the  quality 
assurance  arena  is  the  nominal  group  technique.  This  method  was  originally 
developed  by  Van  de  Ven  and  Delbecq  as  a  tool  for  fact-finding  and  information 
generation. 52  its  adaptation  to  the  derivation  of  quality  assurance  criteria  is 
undocumented  in  the  literature,  although  Williamson  has  used  the  process  to 
identify  quality  assurance  problems  and  set  priorities  for  study. 53  use  0f  the 
technique  involves  the  creation  of  a  highly  structured  environment  in  which 
group  participants  perform  many  of  their  tasks  silently  and  independently,  free 
from  the  possible  influence  of  other  group  members.  The  situation  differs 
significantly  from  the  unstructured  group  environment  in  which  any  group  member, 
by  virtue  of  his  technical  expertise,  organizational  position,  or  dominant  per¬ 
sonality,  may  stifle  the  initiative  and  creativity  of  others. 

Williamson's  application  of  the  nominal  group  technique  involved  the  for¬ 
mulation  of  a  consensus  of  quality  assurance  problem  areas  prioritized  according 
to  perceived  importance.  It  involved  the  listing  of  problem  topics  submitted 
one  at  a  time,  without  criticism,  from  each  panelist  in  round-robin  fashion. 

After  several  rounds  of  topic  submission,  panelists  individually  weighted  each 
topic  on  the  consolidated  listing.  When  weights  for  each  topic  had  been  sunned, 
panelists  were  permitted  to  express  opinions  and  criticisms  concerning  listed 
topics  for  the  first  time.  Panelists  were  then  given  another  chance  to  indivi¬ 
dually  reweight  topics.  Topics  with  the  highest  sunmed  scores  were  considered  to 
be  the  most  important  for  study  prioritization. 54 

Use  of  the  nominal  group  technique  has  several  advantages  over  unstructured 
panel  sessions.  According  to  Jessee,  creativity  is  fostered  in  the  silent, 
independent  generation  of  ideas.  Conformity  and  status  pressure  are  also 
reduced.  Full  participation  by  all  members  is  assured  by  the  round-robin  sub¬ 
mission  of  ideas.  All  opinions  carry  equal  weight,  and  group  members  can 
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disagree  (through  the  independent  weighting  process)  without  openly  arguing. 
Throughout  the  process,  the  expression  of  true  feeling  is  encouraged. 55 

Composition  of  panels 

The  formulation  of  explicit  process  criteria  has  traditionally  been 
accomplished  by  panels  of  "experts,”  or  subspecialists  who  practice  in  the  aca¬ 
demic  environment.  This  procedure  has  been  subjected  to  criticism  in  that  medi¬ 
cal  practice  in  academia  may,  for  any  of  several  reasons,  be  different  than 
practice  in  the  "real  world. "56,57  Donabedian  speculates  that  it  is  perhaps 
because  of  this  criticism  that  current  panelists,  in  addition  to  their  teaching 
affiliations,  must  often  also  be  clinicians  in  active  practice.5** 

The  need  for  a  balance  in  perspectives  in  the  criteria  formulation  process, 
then,  has  not  been  overlooked.  Indeed,  the  potential  contributions  of  those  in 
health  professions  other  than  medicine,  and  even  consumers,  has  been 
recognized.59  The  need  for  incorporation  of  individuals  with  varying 
backgrounds  is  especially  important  to  the  development  of  criteria  to  assess  the 
quality  of  delivery  of  programs  of  health  care  as  opposed  to  the  treatment  of 
specific  diagnoses.  As  an  example,  in  the  development  of  explicit  criteria  for 
evaluation  of  maternity  care,  family  planning,  and  well-child  care  programs  in 
Polk  County,  Florida,  the  process  was  conducted  by  a  twelve-member  multi¬ 
disciplinary  group  consisting  of  staff  nurses,  nursing  supervisors,  administra¬ 
tors,  a  nutritionist,  and  a  physician  clinical  health  officer.60 
Panel  size 

When  the  Delphi  technique  is  used  for  explicit  criteria  development  pur¬ 
poses,  it  can  be  assumed  that  the  collective  opinions  of  a  large  number  of  pane¬ 
lists  are  more  reliable  than  the  opinions  of  a  small  number  (although  there  is 
probably  some  point  at  which  the  degree  of  reliability  ceases  to  increase). 
Hence,  the  larger  the  nunber  of  panelists,  the  better.  This  is  not  the  case, 
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however,  with  face-to-face  panels  where  group  size  tends  to  become  ^manageable 
after  a  certain  point.  A  minimun  panel  size  is,  however,  desirable  to  ensure 
adequacy  of  collective  knowledge  and  experience  among  panelists  and  allow  for 
sufficient  variability  in  viewpoint. 

A  number  of  researchers  have  expressed  opinions  as  to  the  desired  size  of 
panels.  Donabedian  cites  Williamson  as  preferring  panels  of  between  five  and 
thirteen  members  and  Payne  as  favoring  six-person  panels.  Donabedian  himself 
reconmends  a  panel  size  of  six  to  nine  when  utilizing  the  nominal  group  tech¬ 
nique.^1 

A  retrospective  review  of  actual  criteria  formulation  panel  membership 
reveals  panel  size  to  be  rather  small.  Of  a  total  of  41  panels  which  were 
formed,  12,  or  29.3  percent,  were  comprised  of  4  members,  while  14,  or  34.1  per¬ 
cent,  contained  5  members.  Only  8  panels,  or  19.5  percent  of  the  total,  had  a 
membership  of  greater  than  5  persons.  The  range  of  panel  membership  varied  from 
3  to  7  persons. 62 


Objectives  of  this  Study 

1.  To  develop  explicit  criteria  for  assessing  compliance  with  pre¬ 
determined  standards  of  performance  pertaining  to  selected  Army  Occupational 
Health  Program  elements/sub- elements. 

2.  To  determine  whether  developed  criteria  are  realistic,  objective,  and 
measureable . 

3.  To  determine  whether  a  consensus  exists  between  panels  on  the  choice  of 
developed  criteria. 

4.  To  determine  the  degree  to  which  an  installation  occupational  health 
program  is  in  compliance  with  selected  predetermined  standards  of  performance. 

5.  To  determine  whether  the  selected  method  for  evaluating  compliance  with 
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standards  of  performance  is  acceptable  to  an  occupational  health  clinic  staff. 

Criteria  for  Feasible  Solutions 

Testing  should: 

1.  Ascertain  whether  selected  criteria  are  valid  for  evaluating  occupa¬ 
tional  health  program  performance; 

2.  Determine  whether  the  selected  evaluation  method  is  effective  in 
identifying  problems  or  weaknesses  in  an  installation  occupational  health 
program; 

3.  Determine  whether  the  selected  evaluation  method  is  feasible  for  an 
occupational  health  clinic  staff  to  employ. 

Definitions 

1.  The  term  "effective"  refers  to  whether  or  not  problems  or  weaknesses  in 
an  installation  occupational  health  program  can  be  identified  as  a  result  of 
implementation  of  a  given  program  evaluation  methodology. 

2.  The  term  "feasible"  refers  to  acceptability  of  a  proposed  evaluation 
methodology  by  an  occupational  health  service  staff.  Feasibility  is  based  on 
subjective  opinion  as  to  overall  value  of  the  proposed  methodology,  time 
required  on  the  part  of  the  occupational  health  service  staff  to  implement  the 
methodology,  and  any  other  factors  which  an  occupational  health  service  staff 
might  consider  relevant. 


Assumptions 

1.  Standards  of  performance  pertaining  to  Army  Occupational  Health  Program 


elements  and  sub-elements  are  available  to  installation  occupational  health 
service  personnel  for  program  evaluation. 


17 


2.  Problems  which  exist  with  regard  to  occupational  health  program  perfor¬ 
mance  can  be  identified. 

3.  Structural  factors  and  the  process  of  occupational  health  service  deli¬ 
very  are  indicators  of  program  performance  and  the  quality  of  care  provided  to 
installation  employees. 

4.  Concordance  in  the  choice  of  criteria  between  similarly  composed  groups 
of  knowledgeable  individuals  is  an  indicator  of  criteria  validity. 

5.  Criteria  established  by  group  consensus  is  more  valid  in  assessing 
performance  than  individually  -  developed  criteria. 

6.  Professionals  are  more  likely  to  accept  criteria  which  they  themselves 
have  helped  to  formulate. 

7.  Group  -  developed  criteria  are  more  comprehensive  in  scope  when  indi¬ 
viduals  from  more  than  one  professional  discipline  participate  in  the  criteria 
development  process. 


Limitations 

1.  Validity  of  criteria  in  terms  of  contribution  to  outcome  is  not 
addressed  in  this  preventive  health  study  other  than  in  the  assumption  made 
above. 

2.  The  research  effort  is  limited  by  the  cooperation  which  the  researcher 
receives  from  group  participants. 

3.  The  research  effort  is  limited  to  testing  of  the  single  methodology 
selected  for  evaluating  occupational  health  program  performance.  No  alternate 
evaluation  methodologies  will  be  developed  or  tested  during  the  conduct  of  this 
study. 

4.  Researcher  travel  is  limited  to  a  maximum  of  three  trips  to  each  of 


the  following  locations  where  the  research  effort  will  be  focused: 
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a.  Seneca  Army  Depot,  Romulus,  NY. 

b.  U.S.  Army  Environmental  Hygiene  Agency,  Aberdeen  Proving  Ground,  MD 

5.  This  study  constitutes  an  initial  effort  only  and  should  not  be 

regarded  as  the  final  definitive  document  on  this  subject.  Critical  review  of 
final  report  content,  as  well  as  follow-up  activities,  will  hopefully  be  con¬ 
ducted  by  others  knowledgeable  in  the  scope  and  content  of  the  Army  Occupational 
Health  Program. 

Research  Methodology 

This  study  was  conducted  in  the  following  manner: 

1.  A  sample  of  three  Army  Occupational  Health  Program  elements  which  are 
required  by  Army  regulations.  Occupational  Safety  and  Health  Administration 
(OSHA)  standards,  and/or  public  law  were  selected  for  the  study  by  the 
researcher  (See  Annex  A). 

2.  For  each  program  element,  a  standard  of  performance  representative  of 
those  found  in  the  literature  was  selected  by  the  researcher.  Performance  stan¬ 
dards  were  "givens"  and  were  not  open  to  disagreement. 

3.  Criteria  with  which  to  assess  compliance  with  each  given  performance 
standard  were  developed  by  a  panel  of  individuals  assigned  to  the  U.S.  Army 
Health  Clinic,  Seneca  Array  Depot,  Romulus,  New  York.  The  panel  was  composed  of 
one  physician  and  two  occupational  health  nurses.  Each  of  the  three  panelists 
was  first  asked  to  independently  list  as  many  criteria,  in  rank  order  of  impor¬ 
tance,  as  he/she  felt  to  be  necessary  to  assess  compliance  with  each  standard. 
Thereafter,  a  modified  nominal  group  process  (a  structured  technique  used  to 
bring  about  a  group  consensus)  was  employed  in  order  to  develop  consolidated 
lists  of  prioritized  criteria  for  assessing  each  standard. 

4.  Each  panelist  was  also  asked  to  list  at  least  one  measurement  tool  for 
each  of  his/her  original  criteria. 
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5.  Upon  finalization  of  the  three  criteria  lists,  the  researcher  assessed 
compliance  with  each  standard,  using  the  measurement  tools  specified  for  each 
criterion  as  verification  instrunents .  In  this  manner,  problems/weaknesses 
with  respect  to  performance  of  each  program  element  could  be  identified. 
Panelists  were  not  made  aware  of  this  phase  of  the  study  until  after  completion 
of  the  nominal  group  process. 

6.  At  the  conclusion  of  the  criteria  development  portion  of  the  exercise, 
each  panelist  was  given  a  questionnaire  with  which  to  record  his/her  views  on 
this  criteria  development  and  occupational  health  program  evaluation  process. 
Subjective  conclusions  concerning  group  acceptability/process  feasibility  were 
drawn  from  the  answers  provided. 

7.  In  order  to  obtain  some  indication  of  criteria  validity,  the  modified 
nominal  group  criteria  development  process  was  repeated  using  a  similarly  com¬ 
posed  (one  physician  and  two  nurses)  panel  of  experts  currently  assigned  to  the 
Occupational  and  Environmental  Medicine  Division,  U.S.  Army  Environmental 
Hygiene  Agency,  Aberdeen  Proving  Ground,  MD.  Concordance  between  the  two  panels 
in  the  choice  of  assessment  criteria  was  then  determined  by  comparing  the  degree 
of  criteria  agreement  against  a  standard  established  by  the  researcher  after 
careful  scrutiny  of  similar  studies  previously  documented  in  the  literature. 
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II.  DISCUSSION 


Rationale  for  the  Evaluation  Methodology  Selected 

The  approach  used  during  the  conduct  of  this  research  represents  an  attempt 
to  apply  quality  assessment  principles  and  techniques  to  overall  occupational 
health  program  evaluation.  The  method  selected  for  testing  was  chosen  by  the 
researcher  because  it  appears  to  meet  the  needs  for  program  evaluation  which  are 
discussed  in  the  USAEHA.  Occupational  Health  Program  Manual  and  related  literature. 

Quality  assurance,  which  is  generally  construed  to  be  strictly  related  to 
patient  care  activities,  is  only  one  component  of  overall  program  evaluation. 

Given  this  relationship,  it  was  hypothesized  that  the  same  kinds  of  quality 
assessment  activities  which  are  documented  in  the  literature  could  be  expanded 
in  scope  for  program  evaluation  purposes. 

Professionally  developed  quality  assessment  criteria  are  necessary  because 
there  is  no  one  best  or  required  way  of  providing  medical  care  for  any  given 
diagnosis  or  patient  category.  This  situation  is  unlike  that  of  the  Army 
Occupational  Health  Program,  the  elements  of  which  are  specified  to  a  great 
extent  by  laws  and  regulations.  The  inique  aspects  of  each  Army  installation, 
however,  mandate  flexibility  in  the  implementation  of  program  provisions. 
Therefore,  it  was  hypothesized  that  personnel  involved  in  occupational  health 
program  implementation  at  the  local  level  are  in  the  best  position  to  evaluate 
performance  with  regard  to  their  own  programs,  just  as  it  is  generally  assuned 
that  health  professionals  in  any  given  locale  are  most  qualified  to  define  and 
judge  acceptable  quality  of  care  in  their  own  geographical  and  specialty  areas. 

Selection  of  Performance  Standards 

As  is  evident  from  a  review  of  Table  1,  a  total  of  sixteen  program  elements 
(several  of  which  are  divided  into  sub-elements)  are  included  as  components  of 
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the  Army  Occupational  Health  Program.  In  order  to  determine  the  degree  to  which 
program  elements/sub-elements  are  being  carried  out,  it  was  first  necessary  to 
establish  performance  standards  for  each.  These  standards  were  based  on  the 
content  of  applicable  laws  or  regulations.  Performance  standards,  then,  must  be 
externally  imposed  upon  the  installation  rather  than  being  internally  deter¬ 
mined.  The  imposition  of  broadly-worded  standards,  however,  permits  considerable 
latitude  on  the  part  of  installation  personnel  in  their  approaches  to 
compliance. 

In  order  to  test  the  selected  evaluation  methodology,  performance  standards 
pertaining  to  two  Army  Occupational  Health  Program  elements  and  one  sub- element 
were  selected  by  the  researcher.  These  are  listed  with  their  sources  in  Table  1. 


TABLE  I 


Selected  Occupational  Health  Program  Performance  Standards 
Program  Element/ 

Sub -Element  Performance  Standard  Source 


Preplacement 

examinations 


Preplacement  examinations 
which  are  sufficiently 
inclusive  to  aid  in 
suitable  job  placement 
will  be  provided  whenever 
medical  evaluation  is 
necessary  for  work 
assignment. 


HSC  Pamphlet 
40-2;  Standards, 
Interpretations  and 
Audit  Criteria  for~ 
Performance  of 
Occupational  Health 
Programs? 


Health  Education 
and  Counseling 


There  shall  be  a  health 
education  and  counseling 
program  to  promote  both 
general  health  maintenance 
and  safe,  healthful  work 
practices. 


Standards,  Interpret¬ 
ations  f  and  Audit 
Criteria  for  Perform¬ 
ance  of  Occupational 
Health  Programs. 


Pregnancy  Surveillance  Essential  health  supervision  AR  40-5. 

Program  shall  be  provided  to  ensure 

that  women  workers  or  their 
pregnancies  are  not  adversely 
affected  by  their  work  assign¬ 
ments. 
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It  should  be  noted  that  only  one  of  the  above  standards  (pertaining  to 
health  education  and  counseling)  was  taken  verbatim  from  its  source.  The  stan¬ 
dard  for  preplacement  examinations  represents  a  combination  of  two  statements 
taken  from  the  sources  shown.  The  pregnancy  surveillance  standard  is  a 
rewording  of  a  similar  statement  contained  in  AR  40-5.  The  precise  wording  of 
the  selected  standards  could,  in  reality,  be  subject  to  some  debate.  For  this 
reason,  and  because  this  research  was  not  intended  to  assess  the  appropriateness 
of  program  elements  or  performance  standards,  acceptance  of  the  selected  stan¬ 
dards  as  written  was  made  a  condition  of  the  study. 

Rationale  for  Selection  of  Seneca  Army  Depot  as  a  Site  for  this  Study 

Seneca  Army  Depot  (SEAD),  Romulus,  New  York,  was  selected  as  the  study  site 
for  this  project  for  several  reasons.  First,  a  major  portion  of  the  health  cli¬ 
nic's  workload  is  occupational  health  oriented.  Approximately  900  civilians  and 
500  military  personnel  are  assigned  at  the  installation  in  various  jobs  related 
to  weapons  maintenance,  ammunition  storage  and  demilitarization,  strategic 
materials  stockpiling,  and  industrial  plant  equipment  refabrication. 

Second,  the  health  clinic  staff  consists  of  two  physicians  and  two  nurses 
who  were  available  for  study  participation  as  panelists.  The  researcher  was 
desirous  of  having  at  least  four  individuals  on  the  criteria  development  panel 
in  order  to  conform  with  recommendations  previously  cited  from  the  literature. 

In  addition,  this  group  was  previously  observed  to  work  well  together.  The 
importance  of  this  factor  in  the  face-to-face  criteria  development  process  was 
noted  by  Williamson. 1 

The  final  reason  for  the  selection  of  SEAD  as  a  study  site  was  the  interest 
in  the  study  shown  by  the  health  clinic  officer- in-charge  and  his  familiarity 
with  the  intent  of  the  quality  assurance  process  as  espoused  by  the  JCAH.  The 
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researcher  was  aware  of  the  difficulties  involved  in  obtaining  cooperation  for  a 
study  of  this  nature.  Therefore,  the  interest  expressed  by  study  participants 
was  an  important  criterion  for  site  selection. 

Conduct  of  the  Research  at  Seneca  Army  Depot 

Preliminaries 

Williamson,  in  his  research  effort  to  formulate  priorities  for  quality 
assurance  activities  using  a  modified  nominal  group  technique,  held  a  training 
session  approximately  one  month  prior  to  the  formal  priority  development 
meeting.  This  session  served  to  familiarize  participants  with  the  panel  tech¬ 
nique  to  be  used  and  stimulate  ideas  for  subsequent  presentation. 2 

A  similar  methodology  was  used  in  this  study.  Approximately  two  months 
prior  to  the  criteria  development  session,  the  researcher  visited  the  SEAD 
health  clinic  for  the  purpose  of  explaining  the  intent  and  proposed  conduct  of 
the  study  and  soliciting  support  for  his  effort.  Specific  standards  for 
assessment  were  not  presented  to  the  panelists  at  this  time,  nor  was  the 
criteria  development  technique  to  be  used  demonstrated.  The  process  to  be 
followed  was,  however,  explained  and  questions  answered.  Despite  initial  oppo¬ 
sition  from  one  of  the  prospective  panelists  who  was  concerned  about  potential 
criticism  of  his  medical  practice,  group  cooperation  was  secured. 

The  next  preliminary  phase  involved  the  preparation  and  mailing  of  a  letter 
of  instruction  to  each  SEAD  panel  participant.  A  copy  of  this  letter  and  its 
inclosures  are  attached  as  Appendix  B.  The  criteria  development  process  which 
was  described  in  the  letter  was  similar  to  the  one  used  by  Williamson,  with 
several  exceptions.  First,  as  a  time-saving  measure,  panelists  were  asked  to 
have  their  initial  criteria  lists  prepared  in  advance  of  the  researcher’s  arri¬ 
val  for  the  panel  session.  This  procedure  would  permit  the  advance  preparation 
and  immediate  availability  of  first-round  consolidated  criteria  lists  at  the 
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outset  of  the  panel  discussion.  Second,  panelists  were  asked  to  initially  list 
criteria  for  assessing  compliance  with  each  standard  in  order  of  perceived 
importance,  with  the  most  important  criterion  being  listed  first.  The  defini¬ 
tion  of  "important"  was  left  up  to  each  panelist. 

The  letter  of  instruction  also  emphasized  the  need  for  criteria  realism, 
objectivity,  and  measureability,  and  defined  each.  The  requirement  for  the 
selection  of  at  least  one  measurement  tool  for  assessing  compliance  with  each 
listed  criterion  was  intended  to  serve  a  dual  purpose;  first,  to  ensure  cri¬ 
terion  measurability,  and  second,  to  provide  the  researcher  with  a  means  of 
assessing  the  extent  of  compliance  with  a  given  criterion  in  order  to  determine 
shortcomings  in  overall  program  performance.  Panelists  were  not  informed  in 
advance  that  the  measurement  tools  which  they  provided  would  be  used  to  assess 
compliance  with  their  own  selected  criteria.  To  do  so  would  have  undoubtedly 
biased  the  criteria  selection  process. 

Development  of  final  consolidated  criteria  lists 

Each  of  the  three  face-to-face  panel  sessions  began  with  the  distribution 
of  an  initial  consolidated  criteria  list  for  assessing  compliance  with  one  of 
the  selected  standards.  Panelists  were  permitted  time  to  review  the  list  and 
ask  clarification-type  questions  about  the  criteria.  Criticism  of  criteria  was 
not  permitted  at  this  point.  Panelists  were  also  given  the  opportunity  to  add 
overlooked  criteria  to  the  consolidated  list. 

Upon  conclusion  of  the  initial  review,  panelists  were  asked  to  rank  each 
criterion  on  the  consolidated  list,  again  in  order  of  perceived  importance.  The 
most  important  criterion  was  to  be  ranked  "1,"  and  so  on.  When  this  process 
was  completed,  the  researcher  collected  the  ranked  lists,  sunned  the  scores 
given  to  each  criterion,  and  prepared  a  new  consolidated  criteria  listing  with 
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criteria  ranked  according  to  sunned  scores. 

The  second-round  criteria  list  was  distributed  to  panelists  for  review  and 
discussion.  At  this  point,  opinions  and  criticisms  pertaining  to  criteria  were 
allowed. 

After  discussion  had  been  completed,  criteria  were  once  more  re-ranked  by 
panelists.  The  scores  awarded  to  each  criterion  were  again  summed  and  a  final 
consensus  criteria  list  was  prepared.  For  research  purposes,  content  of  each 
final  consolidated  criteria  list  was  limited  to  the  ten  criteria  most  highly 
ranked  by  panelists. 

Review  of  measurement  tools 

Measurement  tools  previously  selected  for  verifying  compliance  with  each  of 
the  criteria  on  the  final  consolidated  criteria  lists  were  reviewed  and 
clarified  for  the  researcher.  In  same  cases ,  additional  measurement  tools  were 
added  to  the  list.  Upon  completion  of  this  procedure,  panelists  were  informed 
of  the  researcher's  intention  to  use  listed  measurement  tools  as  a  basis  for 
assessing  compliance  with  listed  criteria,  a  task  which  was  subsequently 
accomplished. 

Completion  of  quest ionaires 

Each  panelist  was  given  a  questionnaire  with  which  to  record  his  views  con¬ 
cerning  the  criteria  development  process  in  which  he  had  participated  (See 
Appendix  C).  Questionnaire  results  were  used  to  evaluate  acceptability/feasbility 
of  the  evaluation  methodology  which  was  tested. 

Additional  comments 

The  panel  which  met  to  develop  criteria  lists  and  measurement  tools  was 
composed  of  one  physician  and  two  nurses.  The  second  physician,  who  had 
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originally  agreed  to  assist  in  the  study,  was  hospitalized  and  underwent  bypass 
surgery  several  weeks  prior  to  the  panel  phase.  Therefore,  this  individual  was 
unavailable  for  participation. 

Development  and  ranking  of  criteria  lists  and  measurement  tools  by  pane¬ 
lists  was  conducted  as  described  in  the  study  protocol.  An  unanticipated 
question  was  raised,  however,  as  to  the  procedure  for  ranking  essentially  iden¬ 
tical  criteria  submitted  by  different  panelists.  Not  wanting  to  bias  results  by 
arbitrarily  eliminating  duplicate  criteria,  the  researcher  instructed  panelists 
to  rank  one  criterion  according  to  their  perceptions  of  its  true  relative  impor¬ 
tance  while  assigning  a  lower  score  to  each  similar  item.  This  procedure  worked 
well  because  panelists  were  able  to  agree  upon  which  duplicate  criteria  to  rank 
high  and  low  during  the  discussion  phase  preceeding  the  final  independent  ranking 
process . 

The  measurement  tools  selected  to  verify  compliance  with  self -developed 
criteria  were  utilized  by  the  researcher  to  assess  overall  compliance  with  each 
standard.  Verification  was  conducted  largely  through  personal  observation, 
interviews  with  the  health  clinic  staff,  and  reviews  of  medical  records. 

Panelists  were  surprised  and  expressed  some  opposition  to  the  conduct  of  this 
portion  of  the  study.  Initial  disagreement  was  somewhat  lessened,  however,  when 
the  reason  for  not  originally  disclosing  the  researcher's  intentions  was 
explained  to  the  panel. 

Results  of  the  Research  Conducted  at  Seneca  Army  Depot 

Tables  2,  3  and  4  summarize  the  results  obtained  during  the  SEAD  portion  of 
this  study.  Included  are  consolidated  criteria  lists  and  measurement  tools 
selected  to  assess  compliance  with  each  standard  and  a  listing  of  problems/ 
weaknesses  identified  during  the  compliance  verification  process. 
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8.  Proper  equipment  is  on  a.  Examination  requirements  for  specific  a.  Workers  exposed  to 

hand  at  the  health  clinic  for  occupations  as  contained  in  the  publications  radiation  require  slit 

the  conduct  of  required  listed  in  6  a.  above;  lamp  testing.  Equipment 
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The  content  of  Tables  2,  3  and  4  clearly  shows  that  the  criteria  development 
methodology  employed  in  this  study  was  effective  in  producing  explicit  structure 
and  process  criteria  for  assessing  compliance  with  preselected  performance 
standards.  The  ranking  process  was  highly  effective  in  eliminating  those  cri¬ 
teria  which  were  redundant  or  of  little  perceived  importance.  Although  some 
disagreement  was  expressed  by  panelists  concerning  the  final  rankings  for  each 
list,  only  the  relative  importance  of  criteria  were  of  concern  as  opposed  to 
final  criteria  list  content.  It  was  obvious,  then,  that  this  modified  nominal 
group  procedure,  when  utilized  as  described  herein,  is  effective  as  a  process  to 
'Veed  outM  criteria  which  are  perceived  to  be  less  essential  while  retaining 
those  with  the  greatest  value  for  compliance  assessment  purposes. 

The  requirements  for  criteria  realism,  objectivity,  and  measureability,  as 
expressed  to  panelists  in  the  letter  of  instruction,  were  largely  satisfied 
with  one  exception.  A  pregnancy  surveillance  program  criterion,  which  required 
identification  of  females  who  may  become  pregnant  (number  two)  was  determined  to 
be  unrealistic  and  not  measureable.  In  addition,  the  measurement  tool  for 
assessing  compliance  with  criterion  nunber  one  on  the  same  list  was  deemed 
to  be  inappropriate.  One  other  inadequate  measurement  tool  was  also  noted. 
Assessment  of  the  final  "preeraployment  physical  examination”  criterion  could  not 
be  accomplished  using  the  measurement  tool  provided.  Indication  for  referral  to 
a  personal  physician  is  a  matter  of  professional  judgement;  therefore,  only 
through  the  peer  review  process  can  compliance  with  this  criterion  be 
appropriately  assessed. 

The  existence  of  problems  or  weaknesses  in  meeting  criteria  indicates  the 
extent  of  compliance  with  predetermined  performance  standards.  Table  5  sum¬ 
marizes  the  results  of  the  compliance  verification  process: 
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Table  5 

Sunmary  of  Results  from  Compliance  Verification 

Process 

Program  Element/ 
Sub -Elements 

Total  Ntmber 
of  Measureable 
Criteria 

Nunber  of  Criteria 
in  which  Problems/ 
Weaknesses  were  Noted 

Per  Cent  of  Criteria 
in  which  Problems/ 
Weaknesses  were  Noted 

Preemployment 

Physical 

Examinations 

10 

3* 

30 

Health  Education 
and  Counseling 

10 

6 

60 

Pregnancy  7 

Surveillance  Program 

4** 

57 

♦Includes  criterion  for  which  selected  measurement  tool  is  inadequate  for 
compliance  assessment. 


♦♦Includes  criterion  for  which  selected  measurement  tool  is  inappropriate  for 
compliance  assessment. 


It  is  apparent  from  the  above  results  and  more  detailed  observations  con¬ 
tained  in  Tables  2,  3  and  4  that  a  variance  exists  between  what  panelists  say 
should  be  done  and  what  actually  done.  This  finding  is  comparable  to  results 
obtained  in  a  similar  study  conducted  by  Wagner,  et  al.  which  measured  concor¬ 
dance  between  physician  opinion  and  recorded  clinical  practice. 3 

An  examination  of  questionnaire  comments  submitted  by  the  three  SEAD  pane¬ 
lists  revealed  both  a  great  reliance  on  personal  knowledge  of  occupational 
health  programs  during  the  initial  development  of  criteria  lists  and  a  general 
dissatisfaction  with  the  study  methodology  employed.  Responses  provided  to  the 
first  question  indicated  only  minimal  consultation  with  available  Army 
Occupational  Health  Program  references,  even  though  several  were  available  to 
health  clinic  employees.  The  failure  to  utilize  available  references  may  indi¬ 
cate  an  initial  lack  of  motivation  to  assist  in  the  study  or  may  be  reflective  of 
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the  lack  of  preparation  time  available  to  panel  participants  due  to  the  absence 
of  one  physician  and  the  heavy  clinic  workload.  The  researcher  tends  toward  the 
latter  explanation  in  light  of  additional  comments  that  the  criteria  development 
process  was  too  time  consulting,  laborious,  tedious,  and  caused  panelists  to  get 
behind  in  their  work. 

In  retrospect,  question  three,  which  concerned  the  value  of  the  study,  may 
have  been  inappropriate  since  the  panelists,  although  appraised  of  the  overall 
research  objective  of  identifying  program  problems/weaknesses,  were  asked  this 
question  prior  to  the  conduct  of  the  compliance  verification  process  by  the 
researcher.  If  they  had  been  aware  of  the  results  obtained  during  this  study 
phase,  their  comments,  which  were  largely  negative,  might  have  been  more 
favorable.  It  was  interesting  to  note  that  one  panelist,  after  providing  a 
negative  comment  to  this  question,  went  on  to  state  that  "no  quality  controls" 
existed  for  the  SEAD  Occupational  Health  Program. 

Answers  provided  to  the  fourth  question  revealed  a  diversity  of  opinion  on 
how  best  to  evaluate  an  installation  occupational  health  program.  One  panelist 
suggested  a  review  of  occupational  illnesses,  injuries,  and  employee  sick  leave 
records.  This  approach  focuses  on  acute  episodes  and  completely  ignores  eva¬ 
luation  of  program  effectiveness  in  terms  of  chronic,  insidious  disease  and 
injury  (such  as  hearing  loss)  prevention.  Another  panelist  stated  that  specific 
evaluation  requirements  should  be  passed  down  from  higher  levels  rather  than 
being  formulated  at  SEAD.  In  researcher's  opinion,  this  comment  again  reflects 
the  lack  of  time  available  to  the  health  clinic  staff  to  "sit  down"  and  objec¬ 
tively  attempt  to  evaluate  their  program.  It  may  also  indicate  a  lack  of  desire 
to  do  so  in  the  absence  of  any  formally  stated  requirement  from  higher  headquar¬ 


ters. 
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Conduct  of  the  Research  at  the  U.S.  Army  Environmental  Hygiene  Agency 

Because  of  local  differences  in  the  implementation  of  the  Army  Occupational 
Health  Program  and  the  composition  of  occupational  health  service  staffs,  it 
would  be  expected  that  criteria  developed  to  assess  compliance  with  performance 
standards  would  vary  somewhat  between  installations.  Because  of  these  and 
perhaps  other  variables,  it  would  be  difficult  to  assess  criteria  validity 
through  a  comparison  of  criteria  lists  developed  at  different  installations, 
unless  the  installations  were  carefully  matched.  Even  then,  reproducibility  of 
structural  and  process  criteria  would  only  provide  an  indication  of  true  vali¬ 
dity,  as  previously  discussed. 

In  an  attempt  to  gain  a  basic  indication  of  SEAD  criteria  validity,  the 
same  criteria  development  process  that  was  conducted  at  SEAD  was  carried  out 
using  a  panel  of  occupational  health  experts  assigned  to  the  Occupational  and 
Environmental  Medicine  Division,  U.S.  Army  Environmental  Hygiene  Agency  (AEHA), 
Aberdeen  Proving  Ground,  Maryland.  Personnel  assigned  to  this  division  are 
routinely  tasked  with  monitoring  installation  compliance  with  applicable  occupa¬ 
tional  health  laws  and  regulations.  They  are  therefore  able  to  provide  highly 
educated  opinions  as  to  the  selection  of  criteria  for  compliance  assessment 
purposes. 

The  objective  of  this  exercise  was  to  determine  the  extent  of  agreement,  or 
concordance,  of  criteria  selection  between  the  two  panels.  As  was  the  case  at 
SEAD,  the  methodology  employed  involved  a  preliminary  visit  to  AEHA  to  explain 
study  intent  and  conduct  and  to  secure  support.  This  visit  was  followed  by  a 
letter  of  instruction  similar  to  that  sent  to  SEAD  panel  members  (Appendix  D). 

The  original  AEHA  panel  was  to  consist  of  two  physicians  and  two  nurses;  one  of 
the  physicians,  however,  was  unable  to  participate  due  to  a  last-minute  tem¬ 
porary  duty  assignment.  Final  panel  membership,  then,  consisted  of  one  physician 
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and  two  nurses,  a  composition  which  matched  that  of  the  SEAD  panel. 

Development  of  criteria  lists  by  panelists  was  carried  out  according  to  the 
protocol  contained  in  the  letter  of  instruction.  The  provision  of  measurement 
tools  was  again  requested  solely  to  ensure  the  measureability  of  each  criterion. 

At  the  conclusion  of  the  criteria  development  process,  panelists  were  asked 
to  complete  the  same  questionnaire  which  was  provided  to  SEAD  participants. 

Results  of  the  Research  Conducted  at  the  U.S.  Army  Environmental  Hygiene  Agency 
Tables  6,  7  and  8  contain  criteria  lists  and  corresponding  measurement 
tools  developed  by  the  AEHA  panelists.  All  criteria  were  determined  to  be 
measureable.  Once  again,  panelists  were  satisfied  with  the  content  of  each 
final  criteria  list. 

Table  9  compares  the  criteria  lists  developed  by  the  SEAD  and  AEHA  panels. 
The  comparison  which  was  made  w?s  based  on  the  reseacher's  opinion  of  criteria 
intent  rather  than  wording  and  is  intended  to  illustrate  the  degree  of  overlap 
in  thinking  among  the  two  panels. 

Table  9 

Comparison  of  Criteria  Lists  -  SEAD  Versus  AEHA 

Program  Element/  Total  Number  Nunber  of  Criteria  Per  Cent  of  Cannon 

Sub-Elements  of  Criteria  Covered  on  Both  Lists  Criteria 


Preemployment  19  12  63 

Physical 

Examinations 

Health  Counseling  19  5  26 

and  Education 

Pregnancy  17*  13  76 

Surveillance 

Program 

*Not  including  one  criterion  which  was  found  to  be  non-measureable. 
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7.  A  health  and  work  history  has  been  obtained  Entry  in  medical  records, 

on  all  pregnant  employees. 

8.  Appropriate  profile  statement  describing  duty  Entry  in  medical  record, 

limitations  has  been  provided  to  all  pregnant 

military  personnel. 
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A  number  of  methods  which  have  been  employed  to  measure  consensus  with 
regard  to  quality  assessment  criteria  have  been  documented  in  the  literature. 
Studies  employing  such  techniques  are  aimed  at  reducing  proposed  criteria  lists 
by  eliminating  criteria  which  are  deemed  nonessential  or  not  relevant.  Various 
standards  or  cutoff  points  for  criteria  inclusion  have  been  established  by 
researchers  based  on  levels  of  agreement  among  experts.  For  example.  Brook 
included  an  explicit  criterion  item  in  his  final  list  if  at  least  two-thirds  or 
more  of  the  judges  said  that  the  item  must  be  performed  or  must  not  be  performed 
in  the  management  of  a  specified  condition. 4  Novick,  et  al.  accepted  only 
those  criteria  which  were  deemed  "relevant”  by  90  per  cent  of  their  study 
participants. 5  Osborne  and  Thompson  considered  a  criterion  to  be  recommended  if 
it  was  rated  "essential"  by  85  per  cent  of  responding  participants. 6  In  a 
fourth  study,  Wagner  et  al.  established  a  65  per  cent  agreement  level  as  indica¬ 
tive  of  criterion  endorsement.7 

Using  a  similar  standard  setting  concept  and  averaging  the  consensus  cutoff 
points  applied  in  the  studies  cited  above,  the  figure  of  75  per  cent  agreement 
on  criteria  contained  in  each  list  was  established  as  a  standard  upon  which  to 
base  an  opinion  of  panel  concordance.  A  review  of  Table  9  data,  then,  would 
indicate  overall  agreement  only  with  respect  to  the  pregnancy  surveillance 
program  element.  The  lack  of  overall  concordance  with  respect  to  the  other  two 
program  elements  raises  a  question  concerning  the  validity  of  these  criteria 
lists. 

Questionnaire  responses  among  AEHA.  panelists  were  more  positive  than  those 
provided  by  SEAD  participants  concerning  the  value  of  the  consensus  criteria 
development  technique  as  an  objective  approach  to  program  evaluation.  The  prac¬ 
ticality  of  the  implementation  of  this  methodology  at  the  installation  level  s 
questioned,  however.  Several  respondents  noted  the  fact  that  numerous 
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installation  occupational  health  clinics  are  staffed  with  only  one  occupational 
health  nurse  and  a  part  time  physician,  a  situation  which  would  presumably  prohi¬ 
bit  the  use  of  this  process.  This  is  a  reasonable  comment,  although  it 
overlooks  the  fact  that  there  are  other  installation  personnel,  such  as  the 
safety  and  civilian  personnel  officers,  who  play  important  roles  in  the  local 
occupational  health  program  and  who  could  theoretically  be  called  upon  to  par¬ 
ticipate  in  criteria  development  efforts. 

AEHA  participants  seemed  to  be  reasonably  satisfied  with  the  occupational 
health  program  evaluation  methodology  which  they  currently  employ.  They  feel 
that  combining  a  pre-survey  questionaire  with  a  subsequent  on-site  verification 
visit  allows  for  adequate  evaluation  at  their  level,  the  purpose  of  which  is  to 
determine  program  compliance  with  legal  and  regulatory  requirements.  Only  one 
respondent  commented  upon  program  evaluation  by  installation  personnel.  This 
individual  expressed  the  need  for  occupational  health  program  evaluation  at 
the  local  level,  but  stated  that  evaluation  tools  and  techniques  should  be  deve¬ 
loped  by  higher  authority  and  passed  down  to  the  installation  occupational 
health  clinic  staff.  The  response  supported  the  similar  opinion  previously  pro¬ 
vided  by  one  of  the  SEAD  panelists. 
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III.  SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS 

Summary 

This  study  was  undertaken  to  test  a  proposed  methodology  for  evaluating  an 
occupational  health  program  at  a  U.S,  Army  installation.  Research  objectives 
and  techniques  employed  were  largely  inspired  by  and  adapted  from  past  and 
current  quality  assurance  literature.  The  methodology  was  examined  from  the 
standpoint  of  validity,  effectiveness  in  identifying  program  problems/weaknesses, 
and  implementation  feasibility.  A  major  assumption  which  was  made  prior  to 
study  initiation  was  that  standards  of  performance  against  which  compliance 
assessment  of  each  Occupational  Health  Program  element  and  sub-element  could  be 
conducted  will  have  been  preselected. 

A  face-to-face  panel  technique  was  employed  to  develop  explicit  criteria 
and  measurement  tools  for  assessing  and  verifying  compliance  with  performance 
standards.  Two  separate  panels  were  formed.  The  first  was  composed  of  Seneca 
Army  Depot  Health  Clinic  staff  members.  The  second  panel  consisted  of  Army 
Occupational  Health  Program  experts  assigned  to  the  U.S.  Army  Environmental 
Hygiene  Agency.  Assessment  criteria  developed  by  each  panel  were  compared  in 
order  to  obtain  an  indication  of  criteria  validity.  Each  member  of  both  panels 
also  completed  a  questionnaire  intended  to  ascertain  the  feasibility  of  utilizing 
the  methodology  tested  for  installation  occupational  health  program  evaluation 
purposes. 

Conclusions 

Criteria  validity 

A  comparison  of  criteria  developed  by  the  two  panels  reveals  a  divergence 


in  thinking  in  several  cases  and  raises  questions  concerning  criteria  validity. 
The  SEAD  final  criteria  list  for  assessing  compliance  with  the  preemployment 
physical  examination  standard  tended  to  focus  on  process  structure  and  mechanics, 
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while  the  AEHA  listing  was  more  oriented  toward  assessing  compliance  with  regu¬ 
latory  requirements.  The  health  education  and  counseling  criteria  lists  also 
revealed  a  difference  in  orientation;  the  SEAD  staff  appeared  to  be  more  con¬ 
cerned  with  the  presentation  of  general  health  education  in  contrast  to  the  AEHA 
panelists,  who  were  more  interested  in  ensuring  the  provision  of  educational  and 
counseling  activities  pertinent  to  identified  workplace  hazards.  Only  in  the 
criteria  lists  developed  to  assess  compliance  with  the  pregnancy  surveillance 
program  standard  was  there  a  concordance  which  surpassed  the  75  per  cent  standard. 

Upon  review  of  the  results  of  this  study,  it  must  be  concluded  that  the 
validity  of  criteria  selected  by  SEAD  panelists  for  assessing  compliance  with 
two  out  of  the  three  given  standards  was  not  demonstrated. 

The  above  conclusion  may  be  interpreted  by  sane  readers  as  being  indicative 
of  different  overall  priorities  which  are  placed  upon  the  delivery  of  occupa¬ 
tional  health  services  by  AEHA  and  installation  personnel,  respectively.  This 
view  may,  in  fact,  be  accurate.  The  researcher  feels,  however,  that  a  generali¬ 
zation  of  this  nature  cannot  be  supported  by  the  results  obtained  from  this 
limited  study,  just  as  a  definitive  statement  confirming  criteria  validity  could 
not  have  been  made  even  if  criteria  concordance  exceeding  75  per  cent  had  been 
observed. 

Identification  of  problems/weaknesses 

Implementation  of  the  test  methodology  was  clearly  effective  in  identifying 
problems/weaknesses  in  the  performance  of  selected  occupational  health  program 
elements/sub-eleraents  .  Installation  health  clinic  staffs  who  utilize  the  test 
methodology  for  problem  identification  purposes  should  be  motivated  to  initiate 
corrective  actions  in  light  of  the  fact  that  the  existence  of  problems/weaknesses 
indicates  noncompliance  with  their  own  criteria. 
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Methodology  feasibility 

In  spite  of  at  least  one  previously-discussed  flaw  in  the  questionnaire 
portion  of  this  study,  conclusions  with  respect  to  the  feasibility  of  imple¬ 
menting  the  test  methodology  were  quite  clear.  On  the  positive  side,  the  process 
was  found  by  the  panel  of  experts  to  be  workable  and  beneficial  for  the  purpose 
of  developing  occupational  health  audit  criteria.  Both  this  group  and  the  SEAD 
participants,  however,  felt  the  method  to  be  impractical  because  of  a  perception 
that  considerable  time  would  be  required  to  develop  criteria  and  because  of  a 
lack  of  personnel  at  some  installations  to  serve  on  criteria  development  panels. 
There  may  have  also  been  a  tendency  toward  negativism  on  the  part  of  the  SEAD 
staff  because  of  the  possibility  of  being  tasked  with  an  additional  workload 
requirement  if  a  feasible  evaluation  system  were  to  be  developed. 

Overall  findings 

From  the  information  obtained  as  a  result  of  this  study,  it  is  concluded 
that  the  methodology  tested  for  evaluating  an  installation  occupational  health 
program  could  or  would  not  be  successfully  implemented  by  installation  personnel 
at  the  installation  level.  The  methodology  is  not  without  practical  value, 
however.  Recomnendat i ons  for  its  potential  application  are  provided  below. 

Recommendat ions 

1.  Establish  performance  standards  for  each  Army  Occupational  Health  Program 
element/sub-element,  an  action  which  was  assumed  to  have  been  accomplished  pre¬ 
vious  to  the  initiation  of  this  research  project.  AEHA  would  be  the  appropriate 
agency  to  undertake  this  task;  however,  installation-level  occupational  health 
service  personnel  should  be  afforded  the  opportunity  to  comment  on  proposed 
standards  prior  to  their  being  submitted  for  approval  to  Health  Services  Command 
and  Department  of  the  Army. 
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2.  Develop  objective,  realistic,  and  measureable  criteria  in  order  to  assess 
compliance  with  standards  using  the  technique  described  herein.  This  process 
would  be  carried  out  by  panels  of  selected  experts  such  as  those  at  AEHA  and 
reviewed  by  larger  groups  of  qualified  individuals  in  order  to  determine  vali¬ 
dity  through  concordance  measurement. 

3.  Require  installation  occupational  health  service  staffs  to  periodically 
assess  compliance  with  selected  performance  standards  using  developed  criteria. 
Require  assessment  outcomes,  including  problems  noted  and  corrective  actions 
taken,  to  be  periodically  reported,  possibly  via  a  narrative  attached  to  the 
Army  Occupational  Health  Report.  Problems  of  a  quality  assurance  nature  should 
also  be  reported  in  accordance  with  procedures  established  by  the  supporting 
MEDDAC. 

4.  Ensure  that  installation  occupational  health  program  evaluation  activity  is 
made  a  subject  of  interest  during  annual  general  inspections,  especially  in 
locations  where  the  provision  of  occupational  health  services  is  a  primary 
function  of  the  health  clinic. 
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APPENDIX  A 


ARMY  OCCUPATIONAL  HEALTH  PROGRAM  ELEMENTS 


59 


Army  Occupational  Health  Program  Elements 

1.  Inventory  of  occupational  health  hazards. 

2.  Industrial  hygiene  surveys. 

3.  Health  examinations. 

a.  Preplacement  examinations. 

b.  Periodic  job- related  examinations. 

c.  Administrative  examinations  (includes  fitness  for  duty,  return  after 
illness,  and  disability  retirement  examinations). 

d.  Voluntary  health  maintenance  examinations  (includes  specific  disease 
screening  programs). 

4.  Treatment  of  illnesses  and  injuries. 

a.  Job- related. 

b.  Emergency/palliative  treatment  of  nonoccupational  conditions. 

5.  Illness  aosence  monitoring. 

6.  Chronic  disease  or  disability  surveillance. 

7.  Job-related  immunizations. 

8.  Pregnancy  Surveillance  Program. 

9.  Epidemiological  investigations. 

10.  Alcohol  and  Drug  Abuse  Prevention  and  Control  Program. 

11.  Health  education  and  counseling. 

12.  Radiation  Protection  Program. 

13.  Occupational  Vision  Program. 

14.  Hearing  Conservation  Program. 

15.  Safety  and  health  inspections. 

16.  Medical  records  and  reports. 

Sources:  IMted  States  Army  Health  Services  Conmand  Pamplet  11-2,  Occupational 
Health  Program,  July,  1978. 

United  States  Army  Environmental  Hygiene  Agency  Technical  Guide  Nunber 
124,  Occupational  Health  Program  Manual,  1982. 
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APPENDIX  B 


LETTER  OF  INSTRUCTION  TO  CRITERIA  DEVELOPMENT  PANEL  PARTICIPANTS, 
U.S.  ARMY  HEALTH  CLINIC,  SENECA  ARMY  DEPOT,  RC^US,  NEW  YORK 
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REPLY  TO 
ATTENTION  OF 


DEPARTMENT  OF  THE  ARMY 

US  ARMY  MEDICAL  DEPARTMENT  ACTIVITY 
FORT  DEVENS.  MASSACHUSETTS  01433 


29  December  1982 


.  SUBJECT:  Assessment  of  a  Method  for  Evaluating  an  Occupational  Health  Program 
at  an  Army  Installation 


Panel  Participants 
US  Army  Health  Clinic 
Seneca  Army  Depot 
Romulus,  NY  14541 


Dear  Participants: 

Attached  you  will  find  a  document  which  summarizes  the  background  and  method¬ 
ology  for  conduct  of  the  graduate  research  project  about  which  I  spoke  with  each 
of  you  last  month.  Although  I  have  reorganized  my  thinking  somewhat,  the  basic 
concept  and  study  technique  remains  the  same. 

I  request  that  you  review  the  attached  document  and  individually  develop  pre¬ 
liminary  criteria  and  measurement  tool  listings  as  described  in  paragraph  2a-d. 
Feel  free  to  use  any  references  which  will  help  you  to  develop/prior itxze  your 
criteria  lists.  The  remainder  of  the  study  will  be  conducted  when  I  return 

-  _to  the  health  clinic  during  the  last  week  in  January. 

•* 

I  greatly  appreciate  your  interest  and  participation  in  this  study.  Please 
be  aware  that  I  consider  this  to  be  a  very  basic  effort  as  opposed  to  a  re¬ 
search  masterpiece.  At  the  conclusion  of  the  criteria  development  process,  I 
will  pass  out  a  questionaire  in  order  for  you  to  comment  on  the  acceptability 
and  usefullness  of  the  procedure. 

Please  call  if  you  have  any  questions.  I  can  be  reached  at  Au'tovon  256-6894/ 

6806. 


Sincerely , 


1 . Incl 


. MARTIN  J .^FISHER 
MAJ,  MSC 


Administrative  Resident 
US  Army-Baylor  University  Program 
in  Health  Care  Administration 
Fort  Devens  MEDDAC 
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Assess  meet  of  a  Method  for  Evaluating  an 
Occupational  Health  Program  at  an  Army 
I  ratal  la t ion 


In  April  1979,  a  new  quality  assurance  standard  was  adopted  by  the  Joint 
Commission  on  Accreditation  of  Hospitals  (JCAH).  The  standard  mandates  that 
kr«.vn  or  suspected  problems  which  impact  directly  or  indirectly  on  patient  care 
be  identified  and  resolved.  Further  stated  is  the  requirement  to  develop  written 
criteria  which  can  be  used  to  assess  problems  and  measure  compliance  with  achieva¬ 
ble  goals.  Such ' criteria  should  be  acceptable  to  the  clinical  staff  and  should, 
when  applied  to  actual  practice,  be  expected  to  result  in  improved  patient  care/ 
clinical  performance.  AR  40-66,  which  discusses  the  scope  and  content  of  Army 
quality  assurance  activities , additionally  requires  that  criteria  should  be  realis¬ 
tic,  objective,  and  neasureable.  The  term  "realistic"  addresses  the  need  for. 
feasibility  in  meeting  a  given  criterion.  For  example,  it  is  useless  to  write  a 
criterion  which  states  that  an  item  of  information  should  be  computerized  when 
computers  are  not  available.  Objectivity  refers  to  the  necessity  for  a  given 
criterion  to  be  stated  in  terms  sufficiently  precise  to  preclude  variance  in 
interpretation.  One  writer  states  that  the  most  desirable  criterion  is  the  one 
that  requires  an  individual  to  exercise  very  little  judgement  and  answer  its 
implicit  question  with  a  relatively  straight  forward  yes  or  no  response.  Finally, 
measureability  is  defined  as  the  requirement  for  the  answer  to  a  given  criterion's 
implicit  question  to  be  verifiable. 

This  study  constitutes  an  effort  to  apply  the  concepts  discussed  in  the 
JCAH  quality  assurance  standards  to  the  occupational  health  setting  in  order  to 
evaluate  an  installation  occupational  health  program.  The  importance  of  such  an 
evaluation  is  discussed  in  US  Army  Environmental  Hygiene  Agency  Technical  Guide 
.Number  124,  Occupational  Health  Program  Manual.  The  overall  goal  of  the  evalua¬ 
tion  method  to  be  assessed  in  this  study  is  the  identification  of  problems/short¬ 
comings  in  the  delivery  of  occupational  health  services.  Although  your  health 
clinic  nay  never  be  surveyed  by  the  JCAH,  the  Health  Services  Command  Inspector 
General  has  a  significant  interest  in  the  quality  of  services  which  you  provide 
and  the  method  which  you  use  to  evaluate  your  own  program's  performance. 

The  evaluation  method  to  be  assessed  is  based  upon  several  assumptions.  The 
first  of  these  is  that  professionals  are  more  likely  to  accept  criteria  which 
they  themselves  have  helped  to  formulate.  The  second  is  that  criteria  established 
by  group  consensus  are  more  valid  in  assessing  performance  than  individually-de¬ 
veloped  criteria.  Finally,  it  is  presumed  that  group-developed  criteria  will  be 
more  comprehensive  in  scope  if  individuals  from  more  than  one  professional  dis¬ 
cipline  par ticipaDlin  the  development  process. 

/  « 

2.  STUDY  METHODOLOGY: 

a.  For  research  purposes  only,  three  occupational  health  program  elements 
will  be  used  as  the  basis  for  evaluation.  These  elements,  or  sub-programs,  are  re- 
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■v  ny  or  all  of  the  following:  Army  Regulations ,  Occupational  Ssf-.-ty  cnl 
.dministration  ( 0 S ilA )  standards,  or  public  law. 


b.  broad  perforr.an.ee  standards  against  which  the  conduct  of  each  occupational 
health  rrogrem  element  can  be  assessed  will  be  presented  to  you  by  the  researen-r. 
7, are  attached  as  Annex  A.  The  standards  are  either  published  as  stated  cr  are 
reword  inn s  of  program  ob -ectives/purpose  statements  found  in  applicable  Army  Regu¬ 
lations  ,  technical  guides,  and  related  publications. 

c.  On  an  individual  basis,  please  list  criteria  which  you  would  use  to  assess 
compliance  with  each  given  standard.  Try  to  view  criteria  as  a  means  to  evaluate 
installation,  not  just  health  cl  inic,  performance .  Criteria  should  be  written  m 
objective  statements .and  should  be  as  specific  as  possible.  Examples  might  be  the 
following: 

EXAMPLE  1 :  Criterion  for  a  given  Hearing  Conservation  Program  Standard 

"Audiometer  is  calibrated  in  accordance  with  standards  cited  in  TB 
MED  501". 


E XAMT 7 Z  2 :  Criterion  for  a  given  Occupational  Vision  Program  Standard 

"Each  employee  who  has  been  identified  as  working  in  an  eye  hazardous 
area  has  been  issued  safety  glasses". 

There  is  no  limit  to  the  number  of  criteria  which  you  may  use  to  conduct  the 
assessment.  Please  keep  in  mind  the  requirements  for  realism,  objectivity,  and 
measureability . 

d.  In  order  to  ensure  measureability,  please  list  next  to  each  criterion, 
the  measurement  tocl(s)  which  )’ou  would  select  in  order  to  assess  whether  or  not 
that  criterion  is  being  met.  Using  the  previous  examples,  the  following  measure¬ 
ment  tools- might  be  listed: 


EXAMPLE  1: 


Criterion  for  a  given  Hearing  Conservation 
Program  Standard 

"Audiometer  is  calibrated  in  accordance 
with  standards  contained  in  TB  MED  501". 


EXAMPLE  2: 


Criterion  for  a  given  Occupational  Vision 
Program  Standard 

"Each  employee  who  has  been  identified  as 
working  in  an  eye  hazardous  area  has  been 
issued  safety  glasses". 


Measurement  Tool(s) 

1.  TB  MED  501 

2.  Calibration  sticker  on 
Audiometer . 

3.  Most  recent  calibration 
form  in  file. 

Measurement  Tool(s) 

1.  List  of  employees  work¬ 
ing  in  eye  hazardous 
areas . 

2.  Medical  records. 


e.  Once  you  have  developed  criteria  for  assessing  compliance  with  each 
standard,  please  arrange  these  in  order  of  importance,  uTith  the  most  important 
criterion  ranked  number  1.  Each  criterion  must  be  ranked  in  this  manner. 
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f .  I'g-n  arrival  nr  the  hen  I  th  clinic,  the  researcher  will  collect  end  c  : 
sclidctc  1 1  criteria  lists.  One  consolidated  list  of  assessment  criteria  for 
each  standard,  in  rank  order  of  importance,  will  be  prepared  and  distributed  to 
each  participant  at  the  beginning  of  a  face-to-face  panel  session. 

g.  The  purpose  of  the  panel  session  is  to  develop,  for  each  standard,  a 
final  listing  of  no  more  than  the  ten  most  important  assessment  criteria  as  agreed 
upon  by  the  group.  Each  final  list  of  criteria  will  be  selected  as  follows: 

(1)  Panelists  will  first  be  given  an  opportunity  to  individually  review 
the  consolidated,  .prioritized  criteria  list  which  the  researcher  has  prepared. 

If*,  as  a  result  of  this  review,  any  panelist  thinks  of  any  additional  criteria 
which  should  be  added  to  the  consolidated  list,  these  will  be  so  added  with  the- 
priority  established  by  that  panelist.  No  discussion  or  criticism  of  criteria  or 
priorities  will  be  permitted  at  this  point. 

(2)  Time  will  now  be  allotted  for  clarification  of  listed  criteria.  Any 
panelist  can  ask  questions  to  facilitate  his/her  understanding  of  each  criterion. 

As  before,  neither  discussion  nor  judgement  will  be  permitted. 

(3)  Each  panelist  will  now  have  the  opportunity  to  individually  re-weigh 
each  criterion  on  the  master  list.  When  this  has  been  accomplished,  the  researcher 
will  collect  and  collate  the  new  rankings  and  re-list  criteria  in  the  new  order  of 
priority . 

(4)  The  second  ranking  of  criteria  will  now  be  discussed.  Judgements  and 
criticisms  are  allowed  during  this  step  of  the  process. 

(5)  Each  panelist  will  once  again  re-weigh  each  criteria.  This  will  be 
the  final  ranking.  The  ten  (or  fewer  if  total  criteria  are  less  than  ten)  criteria 
with  the  lowest  point  totals  will  be  listed  in  order  of  rank  and  will  be  considered 
to  be  the  panels'  choice  of  assessment  criteria  for  evaluating  compliance  with  the 
given  standard. 

-  _  (6)  The  measurement  tools  which  were  previously  selected  for  assessment  of 

each  criterion  will  now  be  reviewed.  Changes  or  additions  will  be  made  as  deter¬ 
mined  by' panel  members. 

h.  As  a  final  step,  you  will  be  given  a  questionaire  with  which  to  record 
your  views  on  the  criteria  development  and  occupational  health  program  evaluation 
process.  Please  take  time  to  fill  it  out,  as  it  will  be  used  to  evaluate  group 
acceptability  of  the  process  in  which  you  will  have  participated.  . 

3.  Thank  you  in  advance  for  your  interest  and  participation  in  this  study. 


3 
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I.  Pivvrri  I'lcn-.nt:  Preplacement  exanina  t  ions  . 

Standard:  Proplacement  examinations  which  are  sufficiently  inclusive  -to  aid 
suitable  iob  placement  will  be  provided  whenever  medical  evaluation  is  necessary 
for  work  assignment . 


measl'rehett 

ra:;k  criteria  tool  (s) 

1  .* 


2. 


3. 


4. 


6  t 


7. 


8. 


10. 


(Continue  on  reverse  if  necessary) 


in 


II.  ?  v  o  r  r  "■  -  1  •-  a.  o  n  :  :  Health  education  and  counseling. 

Standard:  There  shall  be  a  health  education  and  counseling  program  to  promote 

both  general  health  maintenance  and  safe,  healthful  work  practices. 


CRITERIA 


measure:  :e:;t 

TOOL  (s) 


l. 


3. 


4. 


5. 


6. 


7. 


8. 


9. 


10. 


(Continue  on  reverse 


if  necessary) 
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Criteria  Level  orr-.er.t  and  Occupational  Health 
Program  Ivalention  Questional  re 


'.'.at  resources  did  ycu  use  to  develop  your  initial  lists  of  criteria  end 
r>.::?ur-.t;?n:  tools? 


Chat  is  your  opinion  of  the  process  which  was  used  to  develop  the  final 
consensus  criteria  lists? 


3.  Do  you  feel  that  this  study  was  worthwhile?  Why  or  why  not? 


4.  How  would  you  go  about  evaluating  an  installation  occupational  health  program? 


5.  Other  comments. 


6. 


physician 


nurse 


J 
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APPENDIX  D 


LETTER  OF  INSTRUCTION  TO  CRITERIA  DEVELOPMENT 
PANEL  PARTICIPANTS, 

U.S.  ARMY  ENVIRONMENTAL  HYGIENE  AGENCY 
ABERDEEN  PROVING  GROUND,  MARYLAND 
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DEPARTMENT  OF  THE  ARMY 

US  AFI.’Y  r.'EruSAL  DFPAPTWEMT  ACTIVITY 
;  "  FCPT  SEVENS.  MASSACHUSETTS  01433 

'  -‘LV  '0 
.'.lir.iio:,  Of 


HSXr-O  '  29  December  1?£2 

SUBJECT :  Assessment  of  a  Method  for  Evaluating  an  Occupational  Health  Program 

at  an  A.rmv  Installation 


Panel  Participants 

Occupational  and  Environmental  Medicine  Division 
US  Army  Environmental  Hygiene  Agency 
Aberdeen  Proving  Ground,  MD  21010 


Dear  Participants: 

Attached  you  will  find  a  document  which  summarizes  the  background  and  method¬ 
ology  for  conduct  of  the  graduate  research  project  about  which  I  spoke  with 
each  of  you  last  month.  The  document  has  been  sent  to  study  participants  at 
the  US  Army  Health  Clinic,  Seneca  Army  Depot,  Romulus,  NY,  the  location  which 
has  been  selected  for  on-site  methodology  assessment.  Although  I  have  re¬ 
organized  my  thinking  somewhat,  the  basic  concept  and  study  techniques  remain 
the  same. 

In  order  to  obtain  some  indication  of  criteria  appropriateness  (or  validity), 

I  would  like  to  repeat  the  study  methodology  described  in  paragraph  2  with  a 
panel  of  "experts"  similarly  comprised  of  two  (2)  physicians  and  two  (2)  occ¬ 
upational  health  nurses.  I  would  therefore  request  that  you  review  the  attached 
document  and  individually  develop  preliminary  criteria  and  measurement  tool 
listings  as  described  in  paragraph  2  a-d .  Feel  free  to  use  any  references 
which  will  help  to  develop/prioritize  your  criteria  lists.  In  order  to  mini¬ 
mize  time,  please  mail  your  criteria  lists  to  me  for  consolidation  (as  dis¬ 
cussed  in  paragraph  2e)  prior  to  my- anticipated  24  February  arrival  at  AEHA 
for  the  panel  phase  of  the  study. 

I  greatly  appreciate  your  interest  and  participation  in  this  study.  Please 
be  aware  that  I  consider  this  to  be  a  very  basic  effort  as  opposed  to  a  re¬ 
search  masterpiece.  At  the  conclusion  of  the  criteria  development  process,  I 
will  pass  out  a  questionaire  in  order  for  you  to  comment  on  the  acceptability 
and  usefullness  of  the  procedure. 
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HSHF-0 

SUBJECT :  Assessment  of  a  Method 
at  an  Armv  Installatio: 


29  December  1SS2 

Evaluating  an  Occupational  Health  Program 


Please  call  if  you  have  any  questions.  I  can  be  reached  at  Autovon  256-6894/ 

6  S  06. 


Sincerely , 


1  Incl 


MARTIN  J<£/ FISHER 
MAJ,  MSC 

Administrative  Resident 
US  Army-Baylor  University  Program 
in  Health  Care  Administration 
Fort  Devens  MEDDAC 
Autovon:  256-6894/6806 


74 


Assessment  of  a  Method  for  Evaluating  an 
Occupational  Health  Program  at  an  Army 
Ir.stal  lation 


1 .  BACKGROUND : 

In  April  1979,  a  new  quality  assurance  standard  was  adopted  by  the  Joint 
Commission  on  Accreditation  of  Hospitals  (JCAH).  The  standard  mandates  that 
known  or  suspected  problems  which  impact  directly  or  indirectly  on  patient  care 
be  identified  and  resolved.  Further  stated  is  the  requirement  to  develop  written 
criteria  which  can  be  used  to  assess  problems  and  measure  compliance  with  achieva¬ 
ble  goals.  Such  criteria  should  be  acceptable  to  the  clinical  staff  and  should, 
when  applied  to  actual  practice,  be  expected  to  result  in  improved  patient  care/ 
clinical  performance.  AR  40-66,  which  discusses  the  scope  and  content  of  Army 
quality  assurance  activities , additionally  requires  that  criteria  should  be  realis¬ 
tic,  objective,  and  measureable.  The  term  "realistic"  addresses  the  need  for 
feasibility  in  meeting  a  given  criterion.  For  example,  it  is  useless  to  write  a 
criterion  which  states  that  an  item  of  information  should  be  computerized  when 
computers  are  not  available.  Objectivity  refers  to  the  necessity  for  a  given 
criterion  to  be  stated  in  terms  sufficiently  precise  to  preclude  variance  in 
interpretation.  One  writer  states  that  the  most  desirable  criterion  is  the  one 
that  requires  an  individual  to  exercise  very  little  judgement  and  answer  its 
implicit  question  with  a  relatively  straight  forward  yes  or  no  response.  Finally, 
measureability  is  defined  as  the  requirement  for  the  answer  to  a  given  criterion's 
implicit  question  to  be  verifiable. 

This  study  constitutes  an  effort  to  apply  the  concepts  discussed  in  the 
JCAH  quality  assurance  standards  to  the  occupational  health  setting  in  order  to 
evaluate  an  installation  occupational  health  program.  The  importance  of  such  an 
evaluation  is  discussed  in  US  Army  Environmental  Hygiene  Agency  Technical  Guide 
Number  124,  Occupational  Health  Program  Manual.  The  overall  goal  of  the  evalua¬ 
tion  method  to  be  assessed  in  this  study  is  the  identification  of  problems/ short¬ 
comings  in  the  delivery  of  occupational  health  services.  Although  your  health 
clinic  may  never  be  surveyed  by  the  JCAH,  the  Health  Services  Command  Inspector 
General  has  a  significant  interest  in  the  quality  of  services  which  you  provide 
and  the  method  which  you  use  to  evaluate  your  own  program's  performance. 

The  evaluation  method  to  be  assessed  is  based  upon  several  assumptions.  The 
first  of  these  is  that  professionals  are  more  likely  to  accept  criteria  which 
they  themselves  have  helped  to  formulate.  The  second  is  that  criteria  established 
by  group  consensus  are  more  valid  in  assessing  performance  than  individually-de¬ 
veloped  criteria.  Finally,  it  is  presumed  that  group-developed  criteria  will  be 
more  comprehensive  in  scope  if  individuals  from  more  than  one  professional  dis¬ 
cipline  participacein  the  development  process. 

7.  STUDY  METHODOLOGY: 

a.  For  research  purposes  only,  three  occupational  health  program  elements 
will  be  used  as  the  basis  for  evaluation.  These  elements,  or  sub-programs,  are  re- 
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c  u  i  r  cl  by  any  or  all  of  the  following:  A.rmv  Regulations ,  Group, at  ional  Saf-.-t-  cv.l 
It:.  Administration  ( 0£’:LA)  standards,  or  public  law. 

b.  broad  per forr.ar.ce  standards  against  which  the  conduct  of  each  occupational 
health  program  element  can  be  assessed  will  be  presented  to  you  by  the  resear c n-^r . 
T. are  attached  as  Annex  A.  The  standards  are  either  published  as  stated  cr  are 
r-_wordin.ps  of  program  objectives/purpose  statements  found  in  applicable  Army  Regu¬ 
lations,  technical  guides,  and  related  publications. 

c.  On  an  individual  basis,  please  list  criteria  which  you  would  use  to  assess 
compliance  with  each  given  standard.  Trv  to  view  criteria  as  a  means  to  evaluate 
installation,  not  just  health  cl  inic,  per  forman.ee .  Criteria  should  be  written  in 
objective  statements  and  should  be  as  specific  as  possible.  Examples  might  be  the 
following: 

EXAMPLE-  1 :  Criterion  for  a  given  Hearing  Conservation  Program  Standard 

"Audiometer  is  calibrated  in  accordance  with  standards  cited  in  TB 
MED  501". 

EXAMPLE  2:  Criterion  for  a  given  Occupational  Vision  Program-  Standard 

"Eaan  employee  who  has  been  identified  as  working  in  an  eye  hazardous 
area  has  been  issued  safety  glasses". 

There- is  no  limit  to  the  number  of  criteria  which  you  may  use  to  conduct  the 
assessment.  Please  keep  in  mind  the  requirements  for  realism,  objectivity,  and 
measureability . 


d.  In  order  to  ensure  measureability,  please  list  next  to  each  criterion, 
the  measurement  tool(s)  which  you  would  select  in  order  to  assess  whether  or  not 
that  criterion  is  being  met.  Using  the  previous  examples,  the  following  measure¬ 
ment  tools  jnight  be  listed: 


EXAMPLE  1 :  Criterion  for  a  given  Hearing  Conservation 
Program  Standard 

"Audiometer  is  calibrated  in  accordance 
with  standards  contained  in  TB  MED  501". 


Measurement  Tool(s) 


1.  TB  MED  501 

2.  Calibration  sticker  on 
Audiometer . 

3.  Most  recent  calibration 
form  in  file. 


EXAMPLE  2 :  Criterion  for  a  given  Occupational  Vision 
Program  Standard 

"Each  employee  who  has  been  identified  as 
working  in  an  eye  hazardous  area  has  been 
issued  safety  glasses". 


Measurement  Tool(s) 

1.  List  of  employees  work¬ 
ing  in  eye  hazardous 
areas . 

2.  Medical  records. 


e.  Once  you  have  developed  criteria  for  assessing  compliance  with  each 
standard,  please  arrange  these  in  order  of  importance,  with  the  most  important 
criterion  ranked  number  1.  Each  criterion  must  be  ranked  in  this  manner. 


O 
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arrive!  at  the  health  clinic,  the  researcher  will  collect  and  c-- 
s_iiJare  all  criteria  lists.  One  consolidated  list  of  assessment  criteria  for 
each,  standard,  in  rank  order  of  importance,  will  be  prepared  and  distributed  to 
each  rarticipant  at  the  beginning  of  a  face-to-face  panel  session. 

r.  The  purpose  of  the  panel  session  is  to  develop,  for  each  standard,  a 
final  listing  of  no  more  than  the  ten  most  important  assessment  criteria  as  agreed 
upon  bv  the  group.  Each  final  list  of  criteria  will  be  selected  as  follows: 

(1)  Panelists  will  first  be  given  an  opportunity  to  individually  review 
the  consolidated,  prioritized  criteria  list  which  the  researcher  has  prepared. 

If,  as  a  result  cf  this  review,  any  panelist  thinks  of  any  additional  criteria 
which  should  be  added  to  the  consolidated  list,  these  will  be  so  added  with  the 
priority  established ' by  that  panelist.  No  discussion  or  criticism  of  criteria  or 
priorities  will  be  permitted  at  this  point. 

(2)  Time  will  now  be  allotted  for  clarification  of  listed  criteria.  Any 
panelist  can  ask  questions  to  facilitate  his/her  understanding  of  each  criterion. 

As  before,  neither  discussion  nor  judgement  will  be  permitted. 

(3)  Each  panelist  will  now  have  the  opportunity  to  individually  re-weigh 
each  criterion  on  the  master  list.  When  this  has  been  accomplished,  the  researcher 
will  collect  and  collate  the  new  rankings  and  re-list  criteria  in  the  new  order  of 
priority.  .. 

(A)  The  second  ranking  of  criteria  will  now  be  discussed.  Judgements  and 
criticisms  are  allowed  during  this  step  of  the  process. 

(5)  Each  panelist  will  once  again  re-weigh  each  criteria.  This  will  be 
the  final  ranking.  The  ten  (or  fewer  if  total  criteria  are  less  than  ten)  criteria 
with  the  lowest  point  totals  will  be  listed  in  order  of  rank  and  will  be  considered 
to  be  the  panels’  choice  of  assessment  criteria  for  evaluating  compliance  with  the 
given  standard. 

(6)  The  measurement  tools  which  were  previously  selected  for  assessment  of 
each- criterion  will  now  be  reviewed.  Changes  or  additions  will  be  made  as  deter¬ 
mined  by  panel  members. 

h.  As  a  final  step,  you  will  be  given  a  questionaire  with  which  to  record 
your  views  on  the  criteria  development,  and  occupational  health  program  evaluation 
process.  Please  take  time  to  fill  it  out,  as  it  will  be  used  to  evaluate  group 
acceptability  of  the  process  in  which  you  will  have  participated. 

3.  Thank  you  in  advance  for  your  interest  and  participation  in  this  study. 


3 
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ANNEX  A 


Program  Elements  and  Standards  to  be  Assessed 


*  « 


U1 
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I.  :’v,vrn:i  Element:  Preplacement  examinations. 

Standard :  Pr-splncem-ent  examinations  which  are  sufficiently  inclusive  .to  aid  in 

sui tabic  iob  :  l.icvnent  will  be  provided  whenever  medical  evaluation  is  necessary 
for  werk  a s  s  i yr.'.:  ;n t . 

MEASUREMENT 

RANK  CRilnRIA  I00o  (s) 


1  . 


2. 


3. 


4. 


7. 


8. 


10. 


(Continue  on  reverse  if  necessary) 
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II.  ?r or ram  Element :  Health  education  and  counseling. 

Standard:  There  shall  be  a  health  education  and  counseling  program  to  promote 

both  gen. oral  health  maintenance  and  safe,  healthful  work  practices. 

measurement 

RANK  CRITERIA  TOOL  ( s ) 

1  . 


3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 


(Continue  ~n  reverse  if  necessarv) 
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-  :'.:cv  surveillance  program. 

health  supervision  shall  be  provided  to  ensure  that  v:omen 
<.  ?  are  not  adversely  affected  by  their  work  assignments . 


I  JUKI  A 


( s  J 


ecessarv) 
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